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Navy  Alcohol  and  Drug  Safety  Action  Program: 
RESEARCH  AND  EVALUATION 


The  implicit  goal  of  research  and  evaluation  efforts 
entails  generation  and  dissemination  of  continuous  feedback 
regarding  the  strength  and  integrity  of  each  program 
element,  thus  providing  an  empirically  valid  data  source  to 
compliment  strategic  change,  administrative  decision 
making,  problem  solving,  and  ultimately,  program  evolution. 
The  process  oriented  model  for  evaluation  (see  attached 
Figure)  includes  three  Formative  (namely:  Personnel 
Selection,  Personnel  Training,  and  Process)  and  two 
Summative  (namely:  Participant  Change,  and  System  Impact) 
components  which  represent  fourteen  interrelated  tasks. 

Inclusion  of  the  Formative  components  appears  to  be 
relatively  novel  within  existing  alcohol  and  drug 
prevention/intervention  evaluation  efforts.  These  three 
domains  will  enhance  the  probability  of  identifying  "types" 
of  curricular  elements,  "types"  of  learning  environments, 
and  "types"  of  participants  that  will  contribute  to 
successful  intervention.  Formative  research  and  evaluation 
efforts  will  illuminate  (and  eventually  be  utilized  to 
reduce  and/or  eliminate  contamination)  mediators  of  the 
knowledge,  attitude,  and  behavioral  change  that  the  program 
is  designed  to  accomplish.  By  examining  process,  the 
ability  to  determine  the  relative  contribution  of  each 
component  to  the  complete  intervention  package  will  be 
maximized.  Continuous  monitoring  and  refinement  of  each 
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intervention  component  should,  ultimately,  result  in  a 
"package"  that  is  the  best  possible  for  the  Navy 
population.  The  program  will  be  flexible  to  the  extent 
that  it  is  effective. 

The  Summative  evaluation  goals  represent  "anchors"  for 
all  formative  components;  i.e..  Participant  Change  and 
System  Impact  data  will  be  utilized  to  dismantle  the 
Formative  components,  separate  effective  from  feckless 
pieces  of  the  "package",  strengthen  efficacious  content 
while  simultaneously  altering  or  replacing  ineffective 
content,  and  finally,  to  demonstrate  positive  outcome  to 
Navy,  Civilian,  and  Scientific  communities. 
Instrumentation  for  the  Participant  Change  component 
includes  assessment  of  knowledge,  attitude,  and  behavior. 
System  Impact  is  restricted  to  the  Navy  specified  goals  and 
objectives  which  focus  upon  positive  change  in  terms  of 
non-judicial  punishment  for  alcohol  and  drug  related 
incidents,  work  related  accidents,  sick  days,  hospital 
days,  absenteeism,  rate/rank  reductions,  and  negative 
confrontation  with  either  Naval  or  Civilian  communities. 
Both  Summative  components  are  designed  to  assess  impact,  as 
well  as  providing  a  gauge  for  all  Formative  efforts. 

Collectively,  Formative  and  summative  Components 
combine  to  form  a  solid  foundation  for  a  systematic 
approach  to  research  and  evaluation  which  should 
effectively  reduce  many  of  the  uncontrolled  threats  to 
validity  that  are  quite  obvious  in  any  human  service 
intervention.  The  vast  majority  of  programs  that  resemble 


NADSAP  in  terms  of  delivery  have,  more  often  than  not,  been 
plagued  by  a  host  of  intervening  variables  which  render 
interpretation  of  treatment  effectiveness  impossible. 
Aside  from  obvious  differences  among  program  participants 
(e.g.,  demographic,  social,  psychological,  physiological, 
and  cognitive),  human  service  interventions  introduce 
intra-  and  inter-treatment  variability  as  well.  Potential 
threats  to  validity  and  integrity  of  human  service 
interventions  are  overwhelming;  but,  standardization  within 
and  across  intervention  agents,  measures,  instructional 
techniques,  and  strategies  can  effectively  reduce  invalid 
interpretation  and  inference. 

Essential  to  the  evolution  of  standardized 
intervention  is  a  thorough  understanding  of  all  treatment 
components,  their  relative  contribution  to  the  desired 
outcome  (in  terms  of  the  ratio  of  effort  to  quantity  and 
quality  of  expected  outcome),  and  reduction  and/or 
elimination  of  extraneous  sources  of  variability.  Careful 
research  design  and  thoughtful  conceptual  analysis  are 
prerequisites  for  identification  and  comprehension  of 
intervening  variables.  When  factors  that  mediate  treatment 
(both  intra-  and  extra-treatment)  are  clearly  understood, 
intervention  effects  can  be  traced  throughout  a  network  of 
outcome  measures.  Potential  sources  of  variability 
inherent  to  The  University  of  Arizona  NADSAP  efforts  are 
restricted  to  the  three  Formative  components.  The  specific 


evaluation  tasks  which  constitute  these  general  components 
are  discussed  in  detail  below. 


PERSONNEL  SELECTION 

(1.1  through  1.6) 

Research  and  evaluation  efforts  associated  with 
Personnel  Selection  are  restricted  to  The  University  of 
Arizona  employees:  Trainers,  Site  Coordinators  (Site  level 
program  managers),  Screeners  (Participant  triage  agents), 
and  facilitators  (Intervention  delivery  agents). 
Eventually,  collection  of  continuous  data  from  all  program 
personnel  should  result  in  a  systematic  reduction  in 
uncontrolled  sources  of  variability  (thus  eliminating  many 
of  the  threats  to  internal  and  external  validity)  by 
ensuring  a  high  congruence  between  the  program  as  intended 
and  the  program  as  delivered  (integrity).  Reduction  in 
variability  associated  with  each  of  the  Personnel  Selection 
components  will  occur  as  follows. 

1.1  -  Screener  Screening 

The  primary  function  of  Site  Screeners  is 
preintervention  participant  evaluation  and  triage.  Hence, 
Screeners  are  expected  (by  way  of  a  semistructured 
interview)  to  refer  potential  NADSAP  participants  to  the 
most  appropriate  source  of  help;  be  it  education  (NADSAP), 
or  counseling  (CAAC),  while  simultaneously  providing  a 
positive,  motivating  introduction  to  the  helping  network. 
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Since  the  vast  majority  of  information  which  is  utilized 
for  assessment  and  subsequent  referral  involves  self- 
report,  the  Screener  position  requires  excellent 
communication  skills  (both  verbal  and  nonverbal),  as  well 
as  observation/inference  skills  to  distinguish  accurate  or 
inaccurate  behavioral  accounts.  Screeners  are  expected  to 
track  process  throughout  the  interview,  to  be  nonjudgmental 
during  the  referral  process,  and  to  accurately  articulate 
information  garnered  from  a  "record  review"  and  the 
interview  throughout  the  entire  screening  process. 

Site  Screeners  are  selected  by  Site  Coordinators  on 
the  basis  of  their  skills  in  observation,  tracking, 
communication,  and  decision  making.  Eventually,  Screener 
screening  will  be  contingent  upon  findings  related  to  the 
Participant  Change  component:  Referral  accuracy  will  be 
estimated  from  the  magnitude  of  Participant  change,  and  a 
predictive  model  constructed  from  demographic, 
psychological,  and  skill  characteristics  of  effective  and 
ineffective  on-line  screeners  will  be  constructed  to 
compliment  the  existing  Screener  selection  process. 
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1.2  -  Facilitator  Screening 

Currently,  site  coordinators  screen  prospective 
facilitators  prior  to  The  university  of  Arizona  Training. 
This  screening  process  has  consisted  predominately  of 
structured  interviews  which  were  developed  at  The 
University  of  Arizona.  Although  some  improvement  (as 
reflected  by  an  increase  in  percentage  pass  rates  in  the 
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Facilitator  training  component)  has  been  observed  since 
implementation  of  this  process.  Variation  across  sites  is 
indicative  that  the  observed  benefits  are  not  consistent. 
The  most  obvious  explanation  for  this  inconsistency  is  that 
Site  Coordinators  represent  a  heterogeneous  group  that 
enter  into  the  process  with  a  variety  of  subjective 
impressions  concerning  the  distinction  between  good 
facilitator  prospects  and  inadequate  ones.  The  facilitator 
screening  package  (a  compilation  of  interview  questions, 
paper  and  pencil  instruments,  and  an  "impressions 
inventory"  for  the  Site  Coordinator  was  created  to  reduce 
variability  among  training  groups.  Unfortunately,  the 
degree  of  implementation  during  the  facilitator  screening 
process  varies  considerably  across  Site  Coordinators.  More 
often  than  not,  pass  rates  (during  the  facilitator 
training)  are  reflected  in  the  extent  that  the  facilitator 
screening  package  was  employed  during  the  screening 
process.  Data  indicate  that  the  facilitator  screening 
package  can  effectively  reduce  interindi vidual  variability 
among  perspective  training  groups.  When  the  package  is 
used  consistently  by  all  Site  Coordinators,  facilitator 
variability  within  and  across  all  NADSAP  sites  will  be 
minimized,  thus  enhancing  the  quality  of  The  University  of 
Arizona  facilitator  training.  By  allowing  only  individuals 
with  observable  skills  into  the  training,  training  groups 
will  decrease  in  size  and  pretraining  skill  level  will  be 
amenable  to  more  advanced  training  techniques. 


Initial  efforts  to  streamline  the  facilitator 
screening  package  involved  an  extensive  investigation  of 
the  predictive  relationship  between  facilitator  responses 
to  Rokeach  and  Firo-B  scales  and  their  subsequent  scores  in 
The  University  of  Arizona  Facilitator  Training.  Although 
the  fourteen  subscales  of  these  instruments  accounted  for 
statistically  significant  portions  of  variance  in  trainer 
assessments  of  skill  levels,  the  practical  significance  of 
this  endeavor  was  mediocre  at  best.  Fortunately,  these 
findings  were  interpreted  as  "promising",  and  use  of  the 
Rokeach  and  Firo-B  scales  was  continued. 

Results  from  the  initial  effort  forced  recognition 
that  a  major  problem  associated  with  predicting  trainer 
assessments  of  skill  level  from  facilitator  responses  to 
the  two  instruments  is  related  to  the  error  increase  due  to 
an  interaction  between  unacceptable  intertrainer  agreement 
on  skill  level  and  measurement  error  inherent  to  the  self- 
report  information.  Obviously,  error  originating  from  two 
(both  in  this  instance)  sources  eliminates  the  possibility 
of  accurately  partitioning  the  effects  of  each. 

Accordingly,  a  series  of  psychometric  analyses  were 
initiated  to  examine  the  construct  validity  of  the  Rokeach 
and  Firo-B  scales.  A  new  data  set  was  obtained,  and  factor 
analytic  strategies  were  employed  to  examine  the  factorial 
validity  of  each  instrument.  The  number  of  factors  to  be 
retained  was  forced  to  reflect  the  number  of  constructs 
purportedly  measured  by  each  instrument:  eight  for  the 
Rokeach,  and  six  for  the  Firo-B.  Results  from  this 


■y-4+  ►.  *  .  ‘ rj*  \r.vV.  -■  /  /  s.  f 


8 


analysis  (a  principle  components  extraction  technique, 
followed  by  orthogonal  rotation)  indicated  that  neither  of 
the  instruments  yielded  an  interpretable  factor  structure. 
(If  eigenvalues  greater  than  one  had  been  the  criterion  for 
factor  retention,  the  Rokeach  analysis  would  have  resulted 
in  sixteen  factors,  the  Firo-B  analysis  resulted  in 
thirteen  factors.)  Evidently,  the  Rokeach  and  Firo-B 
instruments  fail  to  possess  the  properties  which  they  have 
been  attributed  in  the  related  literature. 

Failure  to  establish  acceptable  levels  of  construct 
validity  within  either  of  the  predictive  measures 
constituted  the  rationale  for  dismantling  each  scale,  item 
by  item,  in  an  attempt  to  construct  a  new  measure  to 
successfully  predict  trainer  assessments  of  facilitation 
skill  level.  The  lack  of  factor  structure  that  was 
observed  for  both  instruments  provided  convincing  evidence 
that  neither  was  reliable  (and  hence,  by  definition  valid) 
for  predicting  facilitator  success  in  the  training. 

Individual  items  from  both  scales  were  entered 
separately  into  a  regression  analysis  where  trainer 
assessments  of  the  eight  facilitation  skills  represented 
the  dependent  measure.  The  initial  "sifting"  process 
identified  twelve  items  from  the  Rokeach,  and  twenty-two 
from  the  Firo-B  that  combined  to  explain  between  sixty-four 
and  ninety-six  percent  of  the  variation  in  trainer 
evaluations  of  skill  level. 


To  further  refine  the  predictive  accuracy  of  these 
items,  another  analysis  was  computed  to  eliminate  overlap 
between  all  thirty-four  items.  This  procedure  eliminated 
another  ten  items  (two  from  the  Rokeach,  and  eight  from  the 
Firo-B).  The  remaining  twenty-four  items  shared  between 
seventy-eight  and  ninety-nine  percent  of  the  variance  in 
the  dependent  measures.  In  essence,  the  sifting  process 
identified  twenty-four  items  from  an  original  pool  of 
ninety-four,  that  contribute  significant  amounts  of 
predictive  information  to  differentiate  poor  from  good 
facilitation  skills  as  perceived  by  The  University  of 
Arizona  Trainers.  This  process  is  currently  in  a 
replication  phase. 

1.3  -  Site  Coordinator  Screening 

The  Site  Coordinator  (site  Level  Program  Manager) 
functions  as  middle-management;  and  hence,  is  responsible 
for  ensuring  that  the  program  as  conceived  is  congruent 
with  the  intervention  that  Navy  participants  receive. 
Responsibilities  include  Personnel  Selection  (office  staff, 
Site  Screeners,  and  Facilitators),  facilitator  and  class 
scheduling,  and  liaison  between  local  Navy  personnel,  local 
contract  employees,  and  the  central  office  in  Tucson. 

Site  Coordinators  are  selected  by  The  University  of 
Arizona  management  in  accordance  with  their  performance 
during  a  lengthy,  semistructured  interview  (not  drastically 
different  from  the  interview  that  Site  Coordinators  use  to 
select  potential  facilitators  for  the  Training  Component). 
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Skills  associated  with  the  Site  Coordinator  position 


include:  Communication,  Problem  Solving,  Decision  Making, 
and  Management. 

In  an  attempt  to  build  a  statistical  model  to 
discriminate  "effective"  from  "ineffective"  Site 
Coordinators,  all  of  the  on-line  Site  Coordinators 
responded  to  the  Personal  Orientation  Inventory  and  the 
Strong-Campbel 1  State-Trait  Scale.  To  validate  the  utility 
of  these  instruments,  the  principal  investigator, 
operations  coordinator,  and  three  trainers  ranked  each 
respondent  according  to  their  perceptions  of  performance  in 
the  role  of  Site  Coordinator.  Responses  to  the  two 
instruments  were  utilized  to  predict  rank.  Similar  to  our 
experience  with  the  facilitator  screening  package,  results 
from  this  effort  were  unsuccessful.  Future  efforts  will 
employ  different  measures  to  predict  Site  Coordinator 
performance. 

1.4  -  Trainer  Screening 

The  screening  process  for  trainers  consists  of  a 
comprehensive  interview  with  each  program  component 
(namely:  Training,  Operations,  Curriculum,  and 

Evaluation).  The  primary  function  of  this  interview 
process  is  to  screen  out  applicants  who  do  not  have  the 
skills  in  process  training  which  are  deemed  essential  to 
The  University  of  Arizona  philosophy.  Applicants  who 
successfully  complete  the  interview  process  are  hired  (on  a 


probationary  status)  and  given  the  opportunity  to 
demonstrate  their  skills  during  two  separate  trainings  with 
two  different  on-line  trainers.  This  provides  trainer 
applicants  an  opportunity  to  demonstrate  their  skills,  and 
the  on-the-job  performance  evaluation  conducted  by  the  co¬ 
trainers  is  the  ultimate  test  to  determine  whether  or  not 
these  applicants  will  be  selected  as  active  trainers. 

1.5  -  Student  Screening  (Level  1) 

The  goal  of  the  screening  process  is  twofold:  the 
referral  of  Naval  Personnel  to  the  most  appropriate  source 
of  help  and  the  positive,  motivating  introduction  of  the 
individual  into  the  helping  network.  In  The  University  of 
Arizona  program,  this  goal  is  achieved  through  the 
administrative  screening  process. 

The  screening  process  involves  a  meeting  between  the 
Site  Screener  (cf.,  1.3)  and  the  potential  NADSAP 
participant.  During  this  meeting,  the  Navy  member 
completes  the  NAD IS  intake  form  (Navy  Alcohol  and  Drug 
Information  System)  which  elicits  information  regarding 
demographic  and  service  background,  pre-  and  post-service 
legal  history,  and  present  and  past  drug  and  alcohol  use. 
The  Screener  and  Navy  member  review  health  and  service 
records  to  identify  and  discuss  relevant  problems.  The 
Screener's  analysis  of  this  information  is  then  utilized  to 
refer  the  Navy  member  to  the  most  appropriate  source  of 
intervention. 
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The  primary  goal  of  evaluation  and  research  into  this 
particular  component  is  to  ensure  that  all  referral 
decisions  are  accurate,  and  hence,  consistent  across  all 
screeners,  i.e.,  that  any  given  set  of  information  will  be 
interpreted  similarily  by  the  Screener,  and  subsequent 
referral  decisions  will  reflect  this  interpretation  by 
demonstrating  that  any  given  individual  will  be  referred  to 
the  same  level  of  intervention  regardless  of  who  makes  the 
referral.  To  this  end,  the  research  and  evaluation 
component  has  created,  and  distributed  a  "pilot  action 
matrix"  to  facilitate  the  screening  process.  This 
instrument  focuses  upon  factors  which  are  related  to  past 
and  present  alcohol  and  drug  use,  as  well  as  the  magnitude 
and  direction  of  change  evident  in  each  of  the  indicators. 
The  primary  intent  for  this  particular  instrument  is  to 
reduce  variability  between  screener's  assessments  and 
subsequent  referral.  By  assigning  only  those  who  are 
likely  to  benefit  from  the  NADSAP  experience  into  the 
course,  personal  success  and  program  efficacy  will  increase 
substantially. 

1.6  -  Student  Screening  (Level  2) 

The  University  of  Arizona  and  Navy  target  individuals 
who  are  potentially  amenable  to  an  educational  intervention 
only.  Persons  perceived  as  possessing  drug  and  alcohol 
problems  which  extend  beyond  an  educational  intervention 
are  sent,  via  the  Level  1  screening  process  (1.5)  to  Level 
2  screening.  This  process  is  performed  by  Naval  Personnel, 
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and  any  treatment  beyond  the  educational  intervention 
provided  by  The  University  of  Arizona  is  also  a  Navy 
function.  Research  and  evaluation  pertaining  to  Student 
Screening  -  Level  2  is  not  addressed  by  the  current  tasking 
order.  Student  Screening  beyond  Level  1  was  incorporated 
into  the  evaluation  model  for  the  purpose  of  acknowledging 
the  Navy's  continuum  of  care. 


PERSONNEL  TRAINING 

(2.1  through  2.4) 

Personnel  Training  is  provided  by  The  University  of 
Arizona  for  Trainers,  Site  Coordinators,  Screeners,  and 
Facilitators.  Trainers  receive  instruction  in  facilitation 
skills  training  and  observational  techniques  associated 
with  the  skills  which  are  strengthened  during  the 
facilitator  training.  Site  Coordinators,  Screeners,  and 
Facilitators  receive  training  which  is  tailored  to  their 
position.  Evaluation  and  research  concerning  the  training 
components  focus  upon  enhancing  inter-trainer  agreement  on 
facilitation  skill  level,  establishing  the  general izabil ity 
of  skills  observed  in  the  facilitator  training  to  the 
classroom  environment,  and  determining  the  effectiveness  of 
each  training  component  to  overall  program  efficacy.  A 
brief  description  of  research  and  evaluation  efforts 
associated  with  each  of  the  training  components  follows. 
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2.1  -  Screener  Training 

All  University  of  Arizonans  Screeners  are  required  to 
attend  the  twenty-hour  Facilitator  training  which 
emphasizes  eight  facilitation  skills:  Tracking  Content  and 
Process,  Empathy,  Genuineness,  Respect,  Self-Disclosure, 
Openness  to  Feedback,  Giving  Objective  Feedback,  and  Group 
Management.  Since  the  primary  function  of  the  screeners  is 
participant  assessment  and  triage,  skills  which  are  deemed 
important  for  effective  screening  include  communication 
(both  verbal  and  non-verbal),  observation/inference,  and 
the  ability  to  articulate  large  quantities  of  information 
in  order  to  make  the  most  appropriate  referral  decisions. 

Research  and  evaluation  efforts  related  to  this 
component  focus  primarily  upon  standardizing  the  referral 
process  (by  reducing  subjective  interpretation).  In 
essence,  the  screener  must  choose  one  of  two  alternatives 
for  each  potential  NADSAP  participant:  NADSAP  or  CAAC 
(Level  2  screening).  The  "action  matrix"  discussed  above, 
when  used  appropriately,  virtually  eliminates  most 
subjective  interpretation  from  the  process,  however, 
screeners  do  have  the  option  of  selecting  the  opposite 
choice  provided  that  they  provide  in  writing  the  rationale 
for  deviation  from  the  matrix.  To  date,  research  and 
evaluation  regarding  the  Screener  training  component  has 
been  limited  to  matching  screener  decisions  to  "post¬ 
course"  facilitator  perceptions  of  "referral 
appropriateness".  If  facilitators  perceive  the  referral  as 
appropriate  following  their  thirty-six  hours  of  interaction 


with  the  individual,  the  participant  is  returned  to  their 
command  following  NADSAP.  Should  the  facilitator  feel  that 
the  participant  needs  additional  help  following 
participation  in  NADSAP,  she/he  can  refer  them  to  the  CAAC 
for  further  screening  and/or  additional  treatment. 

The  ultimate  goal  of  the  administrative  screening 
procedure  is  to  make  appropriate  referrals  during  the 
initial  intake  interview.  Future  efforts,  following  the 
establishment  of  "Participant  Characteristics"  (cf.,  3.4) 
which  are  correlated  with  positive  change  attributable  to 
participation  in  NADSAP,  involve  incorporating  these 
characteristics  into  the  Screening  process.  Site  Screeners 
will  be  trained  to  utilize  all  relevant  information  prior 
to  the  referral  decision.  Screening  effectiveness  will  be 
established  by  calculating  "hit  rates"  based  upon  all 
dependent  measures  included  in  the  "Participant  Change" 
component  discussed  below. 


2.2  -  Facilitator  Training 

The  University  of  Arizona  Trainers  who  successfully 
complete  the  Trainer  Screening  (1.4)  and  Observation 
Training  (2.4)  meet  prospective  facilitators  who  have 
successfully  completed  the  Facilitator  screening  Process 

(1.2)  during  The  University  of  Arizona  Facilitator  Training 

(2.2) .  This  intensive  twenty-hour  experience  is  designed 
to  develop  and  refine  the  skills  deemed  necessary  for 
effective  facilitation  (namely:  Tracking  Content  and 
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Process,  Respect,  Genuineness,  Empathy,  Openness  to 
Feedback,  Giving  Objective  Feedback,  Group  Management,  and 


Se 1 f -D isc 1 osu r e ) .  During  the  Training  sessions, 
prospective  facilitators  are  given  numerous  opportunities 
to  demonstrate  and  refine  their  skill  level.  Trainers 
observe  process,  evaluate  individual  skill  level,  model 
skills,  and  provide  timely  feedback  to  shape  skills  into 
acceptable  levels. 

Evaluation  of  the  Facilitator  Training  involves  pre- 
and  post-training  assessment  of  facilitator  perceptions 
regarding  the  importance  of  various  characteristics  and 
skills  which  are  conducive  to  NADSAP  participant  knowledge, 
attitude,  and  behavioral  change.  The  self-report 
instrument  is  designed  to  elicit  perceptions  regarding  the 
importance  of  twenty-two  facilitator  skills  and 
characteristics  in  relation  to  effective  facilitation,  as 
well  as  self-assessment  on  each  characteristic.  Ratings  of 
the  importance  of  each  item  are  construed  as  a  measure  of 
Ideal  Facilitation  correlates,  and  self-assessments  are 
conceptualized  as  Real  Facilitation  Ability.  The 
discrepancy  between  Real  and  Ideal  perceptions  is  a 
projective  measure  of  Self-Efficacy  specific  to  each 
training  participant.  Hence,  Training  effectiveness  is 
assessed  from  an  Ideal  perspective  (i.e..  How  does 
participation  in  the  Facilitator  Training  affect  self- 
assessed  facilitation  ability?),  and  a  Self-Efficacy 
perspective  (i.e.,  How  does  participation  in  the 


Facilitator  Training  affect  perceptions  regarding  the 
belief  that  one  is  an  effective  facilitator?). 

Since  all  components  of  the  NADSAP  evaluation  model 
are  inextricably  interdependent/  data  emerging  from  the 
Facilitator  Training  module  can  eventually  be  employed  to 
evaluate  the  effectiveness  of  each  University  of  Arizona 
Trainer  and  each  Facilitator  Training  according  to  the 
three  domains  described  above.  Upon  completion  of  the 
Participant  Change  (4.1  through  4.3)  components/ 
facilitator  effectiveness  can  be  estimated  from  participant 
knowledge/  attitude/  and  behavioral  change/  and 
subsequently  matched  to  the  facilitator's  trainer(s). 
Estimates  of  trainer  effectiveness  will  be  inferred  from 
facilitator  effectiveness  in  the  classroom. 

2.3  -  Site  Coordinator  Training 

The  Site  Coordinator  (Site  Level  manager)  represents 
the  primary  link  between  the  Navy,  The  University  of 
Arizona/  and  the  NADSAP  employees  at  the  Site  level.  Major 
responsibilities  include  scheduling  classes  and 
facilitators/  managing  the  budget/  screening  facilitators 
and  screeners/  and  organizing  and  managing  monthly 
facilitator  meetings. 

Site  Coordinator  Training  consists  of  three  major 
components:  An  orientation  at  The  University  of  Arizona 
following  hire,  periodical  management  assist  visits,  and  an 
Annual  Management  conference.  The  orientation  process 
lasts  approximately  one  week  and  consists  of  an  intense 
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overview  of  each  program  element.  During  orientation,  new 
Site  Coordinators  meet  with  representatives  from  each 
program  component  (namely:  Operations,  Curriculum, 
Training,  and  Evaluation),  and  discuss  issues,  policy,  and 
management  strategies.  The  Annual  Management  conference 
consists  of  five  days  in  Tucson,  and  two  days  in  San  Diego. 
During  the  first  segment,  all  Site  Coordinators  meet  to 
discuss  issues,  exchange  ideas  and  management  strategies, 
and  participate  in  skills  training  in  each  area  relevant  to 
their  position.  The  second  segment  merges  all  contract 
employees  (Site  Coordinators  and  Tucson  staff)  with  Navy 
personnel  from  each  Site.  During  this  segment,  military 
issues  with  contractors,  and  contractor  issues  with  the 
military  are  discussed. 

2.4  -  Observation  Training 

The  University  of  Arizona  Trainers  have  the  final 
input  regarding  who  is,  and  who  is  not  hired  from  the 
prospective  facilitator  pool.  Hence,  the  importance  of 
their  skill  evaluation  ability  cannot  be  overemphasized. 
Since  the  inception  of  this  program,  several  observational 
rating  forms  have  been  developed  and  field  tested  to  ensure 
that  similar  facilitation  skill  levels  will  receive 
identical  assessments,  regardless  of  which  trainer 
evaluates  them.  Currently,  an  eight-item  form  is  being 
used  in  the  field.  The  eight  behavior  skills  (namely: 
Tracking  Content  and  Process,  Empathy,  Respect,  Self- 


Disclosure,  Openness  to  Feedback,  Giving  Objective 
Feedback,  and  Group  Management)  are  evaluated  separately 
for  each  of  the  facilitator  trainees,  and  individual  scores 
from  these  assessments  determine  who  is  able  to  continue 
the  hiring  process  as  a  University  of  Arizona  Facilitator. 

Initial  efforts  involved  extracting  demographic 
information  from  the  completed  employment  forms  retained  in 
the  Tucson  Office.  by  combining  this  information  with  the 
evaluation  scores  assigned  by  The  University  of  Arizona 
Trainers,  we  were  able  to  provide  feedback  to  the  Trainers 
regarding  the  relationship  between  their  skill  assessments 
and  the  specified  characteristics  of  age  and  education 
among  the  Facilitator  pool.  Although  this  information  was 
interesting,  as  well  as  informative,  it  did  not  address 
agreement  issues  associated  with  the  evaluation  process, 
nor  did  it  reflect  the  entire  training  population  (files 
retained  in  the  tucson  office  pertain  to  employees  only. 
The  sample  for  this  investigation  consisted  of  facilitators 
who  had  previously  passed  the  Training  which  necessarily 
meant  that  they  had  received  scores  in  excess  of  three 
across  all  eight  skills). 

In  order  to  better  assess  intertrainer  agreement, 
independent  trainer  ratings  were  mandated  (often  times,  two 
trainers  conduct  the  facilitator  training)  to  pave  the  way 
for  more  valid  and  comprehensive  interpretations  concerning 
the  behavioral  ratings.  A  brief  questionnaire  designed  to 
elicit  demographic  information  from  all  members  of  the 
training  groups  is  also  distributed  during  the  facilitator 


trainings.  The  instrument  contains  questions  regarding 


respondents  gender/  age,  ethnicity,  marital  status,  NADSAP 


background,  and  self-reported  experience  with  alcohol  and 


drugs. 


By  administering  the  questionnaire  at  the  beginning  of 


the  training  experience,  demographic  information  is 


available  on  every  person  in  the  training,  regardless  of 


whether  or  not  the  training  participant  received  passing 


scores  from  the  trainers. 


In  a  related  effort,  an  analogue  study  was  created  to 


obtain  independent  trainer  ratings  of  a  video-taped  segment 


of  an  actual  training.  Training  tapes  were  edited,  and  a 


new  tape  depicting  a  single  facilitator  in  a  role  play  of 


the  Group  Management  exercise  was  developed.  the  final 


version  was  approximately  seven  minutes  in  length. 


Each  of  The  University  of  Arizona  on-line  Trainers  was 


asked  to  view  the  tape,  and  to  assign  behavioral 


assessments  to  each  of  the  eight  facilitation  skills 


presented  on  the  eight  item  feedback  form.  Completion  of 


this  task  represented  the  first  time  that  each  of  the  on¬ 


line  Trainers  had  independently  rated  the  same  target 


person.  From  these  data,  estimates  of  inter-trainer 


agreement  were  established  by  computing  multiple 


correlation  coefficients  for  each  trainer  with  all  others 


for  each  of  the  facilitation  skill  assessments.  These 


results  were  not  particularly  impressive;  R  ranged  from  a 


low  of  .396  to  a  high  of  .727.  Apparently,  the  ambiguity 


inherent  to  each  of  the  eight  facilitation  skill  categories 


induced  the  Trainers  to  define,  and  subsequently  rate  the 


skills  according  to  their  own  priorities  conception  of  the 


specific  behaviors  that  represent  each  of  the  eight  skills. 


In  an  attempt  to  advance  our  understanding  of  the 


process  underlying  the  evaluation  technique,  the  original 


score  matrix  was  inverted,  and  Trainers  were  clustered 


according  to  their  skill  assessment  (Q-factor  analysis). 


This  procedure  yielded  three  independent  factors  (or 


clusters)  of  trainers.  Intertrainer  agreement  within 


factors  ranged  from  R=.739  to  .900  for  factor  one  Trainers, 


and  R=.704  to  .808  for  factor  two  Trainers.  A  single 


Trainer  constituted  factor  three;  within  factor  agreement 


was  neither  computable  nor  interpretable  for  factor  three. 


Although  intertrainer  agreement  within  factors  was 


acceptable,  interfactor  comparisons  were  generally  weak. 


Multiple  correlation  coefficients  calculated  across  trainer 


clusters  (or  factors)  ranged  from  R=.432  to  .727.  Since 


our  ability  to  streamline  an  effective  paper  and  pencil 


screening  tool  for  facilitators  (1.2)  is  inherently  related 


to  the  extent  that  we  can  establish  reliability  within  the 


criterion  measure  (Viz.,  Trainer  ratings  of  facilitation 


skill  level),  the  results  of  the  analogue  study  are  a  step 


toward  this  end. 


Future  efforts  involve  continued  collection  of 


information  regarding  Facilitator  Training  participant 


demographic  characteristics  from  every  training 


participant,  regardless  of  their  performance.  This 
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information,  in  conjunction  with  the  independent  trainer 
evaluations  of  participant  performance  will  yield 
sufficient  data  to  provide  trainers  with  practical  feedback 
concerning  potential  biases  which  are  integrated  into  their 
behavioral  assessment.  This  feedback,  combined  with 
independent  Trainer  ratings  of  the  edited  training  tapes 
(collected  in  a  controlled  environment)  can  then  be 
implemented  to  enhance  intertrainer  reliability.  Immediate 
and  persistent  feedback  should  ultimately  result  in  a 
reduction  of  intertrainer  variability  across  the  behavioral 
ratings.  Furthermore,  the  facilitator  demographic 
information  will  be  employed  to  identify  facilitator 
characteristics  associated  with  (and  predictive  of}  success 
in  the  Facilitator  Training. 

As  noted  earlier,  the  Personnel  Training  component  of 
the  NADSAP  evaluation  model  focuses  primarily  on  reducing 
and/or  eliminating  uncontrolled  variables  inherent  to  the 
overall  intervention  effort.  This  brief  description  of  the 
Personnel  Training  evaluation  tasks  has  highlighted  several 
potential  sources  of  variability,  identified  possible 
strategies  to  reduce  this  variability,  and  presented 
previous  and  proposed  methods  of  decreasing  these  potential 
threats  to  the  integrity  of  the  entire  intervention  effort. 
No  doubt,  much  more  effort  is  required  to  achieve  this 
goal.  The  Model  for  Evaluation  of  NADSAP  provides  insight 
for  increased  ■  nd e r s t and i ng  by  providing  direction  for 
future  efforts. 
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PROCESS 


(3.1  through  3.4) 

While  it  is  recognized  that  standardization  is 
necessary  for  evaluation  of  human  service  intervention 
programs,  it  is  also  evident  that  standardization  is  not 
sufficient  for  program  success.  Hence,  the  rationale  for  a 
process  component  within  the  evaluation  effort.  Even 
though  evaluation  efforts  directed  toward  Personnel 
Selection  and  Personnel  Training  are  reducing  variability 
among  numerous  factors,  not  all  mediating  variables  can  be 
eliminated.  A  thorough  understanding  of  the  intervention 
process  can  enhance  the  effectiveness  of  NADSAP.  As  shown 
in  the  Evaluation  Model,  four  distinct  factors  with 
mediational  potential  have  been  incorporated  into  the 
Process  component.  Obviously,  Facilitator  Characteristics, 
Curriculum,  Classroom  Environment,  and  Participant 
Characteristics  are  Classroom  Environment,  and  Participant 
Characteristics  are  viable  sources  of  program  mediation. 

The  primary  impetus  of  the  Process  evaluation 
component  involves  examination  of  mediating  factors  within 
the  intervention  itself.  Upon  establishing  the  effects  of 
each  factor,  as  well  as  the  interaction  between  these 
factors,  this  knowledge  can  be  employed  to  enhance 
intervention  outcome.  For  example,  in  a  previous 
investigation,  age  of  participant  interacted  with 
facilitator  gender  to  affect  participant  course  and 
facilitator  evaluations.  Younger  participants  possessed 


more  positive  views  of  courses  facilitated  by  female 


facilitators.  Conversely,  as  participant  age  increased, 
evaluations  of  female  facilitated  NADSAP  classes  decreased. 
Should  this  phenomena  emerge  repeatedly  in  future 
investigations,  it  would  behoove  program  administrators  to 
mandate  female  facilitators  for  classes  composed  of  young 
Participants,  and  male  facilitators  for  classes  that  are 
composed  of  older  (30+)  participants.  Ideally,  a  thorough 
understanding  of  all  variables  that  mediate  intervention 
effectiveness  would  guide  manipulation  of  these  factors  to 
maximize  desired  change  in  the  outcome  measures.  The  four 
categories  included  in  the  Process  component  of  evaluation 
are:  Facilitator  Characteristics,  Curriculum,  Classroom 
Environment,  and  Participant  Characteristics. 

3.1  -  Facilitator  Characteristics 

Aside  from  the  Trainer  evaluations  of  skill  level 
during  the  Facilitator  Training,  the  only  feedback  that 
facilitators  receive  is  provided  by  the  NADSAP  participants 
following  the  course.  Participant  perceptions  of 
facilitator  and  course  effectiveness  are  important  to 
evaluation  because  they  are  indicative  of  facilitator 
competency  as  perceived  by  the  target  population. 
Facilitator  Characteristics  (demographic,  cognitive, 
psychological,  and  physiological)  are  probable  mediators  of 
participant  perceptions  of  competency.  Examination  of 
Facilitator  Characteristics,  in  conjunction  with 
Participant  Change,  will  strengthen  the  Facilitator 
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Screening  (1.2)  and  Facilitator  Training  (2.2)  components 
of  the  evaluation  effort.  Furthermore,  insights  for 
molding  the  NADSAP  efforts  around  the  intervention 
population  to  maximize  knowledge,  attitude,  and  behavioral 
change  will  evolve  through  investigation  of  process. 

Measurement  of  knowledge,  attitude,  and  behavior,  both 
pre-  and  post-NADSAP,  reflect  course  effectiveness;  the 
NADSAP  facilitator  and  course  evaluations  reflect 
participant  perceptions  of  course  effectiveness.  The 
difference  between  participant  change  and  perceptions  of 
usefulness  is  of  interest  because  Facilitator 
Characteristics  potentially  mediate  both;  i.e.,  some 
Facilitator  Characteristics  mediate  actual  change,  and 
others  mediate  perceived  change. 

3.2  -  Curriculum 

The  curriculum,  or  plan  of  presentation,  is  composed 
of  several  experiential  and  instructional  components,  each 
accompanied  by  specific  goals  and  objectives.  further, 
each  of  the  modules  can  be  presented  in  a  variety  of 
sequences.  Aside  from  examining  the  relative  effectiveness 
of  each  module,  the  ordering  and  the  sequencing  of 
presentation  are  likely  mediators  of  Participant  Change. 
Comprehensive  experimental  designs  will  be  developed  to 
examine  the  effectiveness  of  each  curricular  component,  as 
well  as  to  establish  the  most  effective  order  and  sequence 
of  presentation.  If  some  modules  are  found  to  be 
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nonef f icacious  according  to  the  desired  knowledge, 
attitude,  and  behavior  change,  they  will  either  be  removed 
or  strengthened  to  contribute  to  the  overall  intervention. 


Likewise,  by  manipulating  order  and  sequence  of 
presentation,  we  can  determine  which  combination  yields 
maximum  change  in  the  outcome  measures. 

3.3  -  Classroom  Environment 

The  educational  literature  is  literally  bloated  with 
studies  which  examine  the  effects  of  environmental 
contingencies  and  the  learning  process.  Educators  and 
psychologists  have  documented  the  impact  of  a  host  of 
environmental  variables  on  subsequent  learning  outcomes. 
Factors  such  as  class  size,  day  versus  night  scheduling, 
location,  length  of  sessions,  and  class  "character"  are 
likely  mediators  of  the  intervention  process.  The 
Classroom  Environment  component  of  the  evaluation  model  was 
included  to  examine  these  variables.  Similar  to  the 
proposed  methodologies  described  above,  experimental 
designs  can  be  utilized  to  investigate  the  effect  of 
variations  in  learning  environments.  This  information, 
will  be  utilized  to  maximize  intervention  effectiveness. 

3.4  -  Participant  Characteristics 

Although  Participant  Characteristics  such  as  age, 
rank,  educational  level,  length  of  time  in  the  service, 
gender,  family  background,  pay  rate,  previous  drug  and 
alcohol  experience,  "quality  of  life",  and  ethnicity  are 
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not  likely  to  be  affected  by  participation  in  NADSAP, 
individual  differences  and  intervention  outcome  can  be 
combined  to  maximize  participant  change.  By  examining 
Participant  Characteristics  and  knowledge/  attitude,  and 
behavior  change  following  NADSAP,  the  mediational  effects 
of  intra-  and  inter-individual  differences  can  be  harnessed 
and  controlled  to  maximize  intervention  effectiveness. 
Previous  studies  which  have  focused  upon  participant 
characteristics  and  participant  perceptions  of  "course 
usefulness"  have  demonstrated  that  an  interaction  does 
exist.  Our  goals  and  objectives  related  to  this  component 
are  inherently  related  to  the  general izabi 1  ity  of  these 
interactions  to  actual  change  in  knowledge,  attitude,  and 
behavior. 

Ideally,  repeated  examination  of  the  relationship 
between  process  variables  inherent  to  the  NADSAP 
intervention  should  provide  a  wealth  of  information 
pertaining  to  Facilitator  Characteristics,  Curriculum 
Modules,  Classroom  Environments,  and  Participant 
Characteristics  which  are  conducive  to  successful 
intervention.  Moreover,  insight  concerning  the  interaction 
between  all  process  variables  can  be  useful  in  "molding" 
the  entire  intervention  package  to  the  target  population  to 
ensure  maximum  change  in  outcome. 


PARTICIPANT  CHANGE 


(4.1  through  4.3) 

The  primary  goal  of  The  University  of  Arizona  NADSAP 
intervention  is  to  change  drug  and  alcohol  related  behavior 
in  the  target  population.  While  knowledge  is  necessary  for 
behavioral  changer  it  alone  is  not  sufficient.  When  the 
Process  components  (3.1  through  3.4)  are  effectively 
manipulated  to  ensure  intervention  effectiveness, 
participant  attitudes  should  change  also.  The  Participant 
Change  components  focus  upon  establishing  the  extent  to 
which  participant  knowledge,  attitude,  and  behavior  are 
altered  as  a  direct  result  of  participation  in  NADSAP. 
Attitudes  of  interest  include  attitudes  toward  alcohol  and 
drug  use,  attitudes  toward  work,  and  most  importantly, 
attitudes  toward  self.  The  University  of  Arizona 
philosophy  is  geared  toward  increasing  self-awareness  and 
responsibility  toward  self  and  others.  The  Participant 
Change  component  of  the  evaluation  model  was  included  to 
investigate  the  extent  that  the  intervention  accomplishes 
these  goals. 

4.1  -  Knowledge  Change 

Knowledge  is  assessed  with  the  forty-item 
questionnaire  developed  by  Malfetti  (1971).  Information 
pertaining  to  scale  construction,  validation  and  related 
psychometric  properties,  and  norms  can  be  obtained  from  The 


4.2  -  Attitude  Change 

The  instrument  for  assessing  attitude  change  consists 
of  a  composite  of  previous1  developed  scales  which  have 
demonstrated  acceptable  reliability  and  validity,  as  well 
as  a  documented  relationship  to  either  alcohol  and  drug 
use,  or  alcohol  and  drug  use/abuse  prevention/intervention. 
Specifically,  modifications  of  the  Rosenbaum  (1980)  Self- 
Control  Scale,  Reynold's  (1982)  version  of  the  Marlow- 
Crowne  (1960)  Social  Desirability  Scale,  Zuckerman's  (1979) 
Sensation  Seeking  scale  (subscales  include  Disinhibition, 
Susceptibility  to  Boredom,  Novel  Experience  Seeking,  and 
Thrill  and  Adventure  Seeking),  the  Rathus  (1973) 
Assertiveness  Inventory,  Rotter's  (1966)  Locus  of  control 
measure,  the  CES-D  Depression  scale  (Radloff,  1977),  Adam's 
measure  of  Ego-Identity  (1983),  and  Rosenberg's  (1965) 
Self-Esteem  Scale. 

4.3  -  Behavior  Change 

Actual  behavioral  measures  are  extracted  from  the  Navy 
Alcohol  and  Drug  Information  System  (NAD I S )  and  the 
participant's  medical  and  personnel  records  during  the 
screening  (1.5).  These  data  include  BAC  at  the  time  of 
arrest  (if  applicable),  pre-service  arrests,  convictions 
and  waivers,  court  martials,  reductions  in  pay,  rate,  and 
rank,  alcohol  and  drug  related  incidents  (traffic  and 
otherwise),  as  well  as  several  self-report  measures  aimed 
at  eliciting  extent  of  alcohol  and  drug  use. 


In  addition,  two  measures  of  drinking  quantity  and 
frequency  are  included  to  examine  intervention 
effectiveness  as  mediated  by  the  extent  of  drinking 
problems.  These  measures  are  the  Alcoholism  Indicator 
Scale  (Malfetti  &  Simon,  1974)  and  the  MAST. 


5.1  -  Formative  Evaluation 

Formative  evaluation  represents  findings  from  each  of 
the  components  which  constitute  the  Personnel  Selection, 
Personnel  Training,  and  Process  evaluation  modules. 
Information  obtained  from  tasks  1.1  through  and  including 
3.4  will  be  "fed  back"  to  all  other  components. 


5.2  -  Summative  Evaluation 

Summative  evaluation  represents  findings  from  each  of 
the  components  which  constitute  the  Participant  Change 
evaluation  module.  Information  obtained  from  tasks  4.1 
through  and  including  4.3  will  be  "fed  back"  to  the  Navy, 
Civilian,  and  Scientific  communities. 


SUMMATIVE  STUDIES-in  progress 

FCP/NADSAP-DUI/DWI/OUI :  two  year  longitudinal  study 

currently  implemented  at: 

San  Diego 
Bremerton 

Data  collection:  record  reviews  and  questionnaires. 

Measurement  intervals:  Pre/Post,  3,6,9,12,18,  &  24 

month  followup. 

Sample  Size  to  date:  241. 

Anticipated  N:  700. 


mm 


Supervisor  Followupl:  post  test  only,  currently  implemented 

at : 

Great  Lakes 

Data  Collection:  questionnaire. 

Measurement  intervals:  3-month  followup. 

Sample  size  to  data:  143. 

Anticipated  N:  -ongoing 


Supervisor  Followup2:  pre/post,  currently  implemented  at: 

San  Diego 
Bremerton 

Data  Collection:  questionnaire. 

Measurement  intervals:  Pre-NADSAP,  and  3-month 

followup. 

Sample  size  to  date:  46 
Anticipated  N:  60 

Participant  Followup:  on  year,  Cross-sectional  study 

currently  implemented  at: 

San  Diego 
Bremerton 
Pearl  Harbor 
Great  Lakes 
Jacksonville 
Yokosuka 

Data  collection:  NADIS,  NADSAP  Client  intake  form,  & 

Questionnaire. 

Measurement  intervals:  NADIS  &  intake  data-pre  NADSAP; 

questionnaires-3,  6,  9,  &  12 
months  following  class 
completion. 

Sample  size  to  date:  119 
Anticipated  N:  1800 

Student  Knowledge  Change:  Pre  and  Post  course  implemented 

at: 

Various  Sites  (n=16) 

Data  Collection:  questionnaire. 

Measurement  interval:  Pre  and  Post  NADSAP. 

Sample  size  to  date:  3126. 

Anticipated  N:  ongoing. 


